When images are to be stored or processed electronically they are almost always spatially sampled as an array of picture elements or pixels. The number of pixels chosen to represent an image will depend on the required spatial resolution, but is usually limited by practical constraints of processing speed, complexity or available data storage and transmission capacity. In television it is common to use different numbers of pixels to represent the luminance of the image and the chrominance of the image. The well-known ITU-R recommendation 601 for sampling television images includes the so-called 4:2:2 sampling structure in which chrominance information is sampled at half the horizontal spatial frequency of the luminance sampling, and chrominance samples are co-sited with horizontally-alternate luminance samples.
It is often necessary to convert a sampled image from one sampling structure to another. Many different television display formats have been developed; conversion between standard-definition and high-definition television formats is a common problem to be solved, and this usually involves changing both the horizontal and the vertical sampling frequency. If such conversions can be made without introducing any distortion of the image, then an image can be processed at any convenient resolution; and, the combination of images having different resolutions, for example in a video editing or production process, is greatly simplified.
International Patent application WO 2009/138804 describes methods of up-sampling image data, so that it is represented by a higher number of samples, and then down-sampling the result to obtain the original samples without loss or distortion. This is achieved by ‘reversible’, finite impulse response (FIR) resampling filters. Suitable filters for conversion between standard-definition and high-definition television formats may have a ‘filter aperture’ of 16 samples. This means that 16 input samples are used to construct each output sample; and, any input sample will contribute to an image region 16 input samples wide. In the case of up-conversion, 16 input samples will correspond to a larger number of output samples. Therefore, the up-converted image will be surrounded by a wide ‘border’ of non-zero samples situated outside the edge of the original image.
This is illustrated in FIG. 1 which shows known up-conversion followed by known down-conversion. For simplicity the Figure shows a filter aperture 12 low-resolution samples wide. Although spatial samples are usually considered mathematically as ‘delta functions’ having infinitely small spatial extent, the array of pixels that represents an image is often represented as a tessellating set of polygons where the centre of each polygon corresponds to a spatial sampling point. Orthogonal sampling structures are typical and the pixels are notionally rectangular; and, in many preferred image sampling systems, they are square. In the figure the samples are represented by rectangles in which the centre of each rectangle corresponds to the sampling point. The figure illustrates one-dimensional sampling, and so the height of the rectangles is arbitrary.
A sequence of low-resolution samples (1) is to be up-converted to a sequence of higher-resolution samples (2), and then reversibly down-converted to a sequence of low-resolution samples (3) that are identical with the original low-resolution samples (1). The samples represent some spatial attribute of an image, for example luminance values for pixels. Let us assume that the sequence of samples represents the start of a television line; and, the line (4) in the figure represents the position of the left-hand edge of the sampled image.
The required sampling frequency and phase of the higher resolution samples (2) is assumed to be arbitrarily defined, and therefore there is no fixed relationship between the positions of low-resolution samples and the positions of high-resolution samples.
If the up-conversion is to be reversed according to the principles described in the above-referenced patent application, then all higher-resolution samples that receive a contribution from an input sample must be available to the down-conversion filter. The higher resolution samples (2) will be created by an FIR up-sampling filter that forms output samples from a weighted sum of input samples that fall within a filter aperture centred on the position of the required output sample. In the figure the up-conversion filter aperture is assumed to be 12 low-resolution samples wide and its position when creating the first higher-resolution output sample is shown by the brace (5). The illustrated position is the earliest position that includes a low-resolution input sample; if the filter aperture were moved one higher-resolution sample pitch to the left, then no input samples would fall within the filter aperture. Thus, in the illustrated example, an additional 9 higher-resolution samples, numbered Yh1 to Yh9 and situated outside the image boundary, have been created. A similar situation arises at the opposite image edge (the right-hand edge, not shown in the Figure) where up-converted samples must continue to be generated until the filter aperture no longer includes any input samples.
We will now consider the down-sampling process. The re-creation of the first lower-resolution sample from the set of higher-resolution samples (2) by an FIR down-sampling filter is shown by the brace (6). According to the teaching of the above referenced patent application, which is hereby incorporated by reference, it is advantageous for the aperture of the down-sampling filter to be identical to the aperture of the up-sampling filter; the figure therefore shows identical aperture widths. As can be seen from the figure, the additional 9 samples lying outside the image are required to reconstruct the first lower-resolution sample of the set (3). And, additional samples will be required to construct samples at the right-hand image edge (not shown in the Figure).
If these ‘border’ samples outside the image area were discarded it would be impossible to reverse the up-sampling without loss or distortion. It is thus necessary to process more samples than the number corresponding to the size of the up-converted image, if reversibility is to be maintained. The number of additional samples depends on the conversion ratio as well as the filter aperture; thus it may not always be possible to know how many additional samples should be stored. In real-time video processing, the number of extra samples may be more than can be processed in the available horizontal and vertical ‘blanking intervals’ between lines and fields or frames.
These difficulties of prior-art resampling systems can be solved by embodiments of the present invention.